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Deconvolving Far-field Optical Images Beyond the Diffraction Limit by 
using Scanned Probe Optical and Non-optical Data as the Constraint in 
Mathematical Constraint Algorithms 



I. Field of the Invention 



The field of the invention is the combination of scanned probe microscopic data 
with far field optical and other images in order to deconvolve these images 
beyond the diffraction limit. 

II. Background of the Invention 

Lens based far-field imaging is limited in the resolution that it can achieve by 
the characteristics of the lens. In general there are the problems of diffraction 
of the lens, the problems with aberration of the lens and the problems of 
out-of-focus radiation.The latter problem is generally partially improved by the 
use of confocal imaging methodologies or, in optics, also non-linear imaging 
techniques are useful, while the solution of the former problems are partially 
addressed by measuring the point spread function of the lens and then using 
computer deconvolution to remove these effects from the image. Even the later 
problem can be addressed without confocal or non-linear imaging by 
considering both the in-focus and the out-of-focus point spread function and 
using deconvolution routines to try and eliminate these effects. Numerous 
algorithms have been devised to address these problems of computer 
deconvolution of far-field imaging data but none are completely successful and 
none of them have the ability to carry the far-field image to the realm beyond 
the diffraction limit as defined for example by the Rayleigh criterion, which is 
approximately 1/2 of the wavelength of the radiation that is being used or for 
visible 500 nm light this is 250 nm . 

In terms of deconvolution algorithms, a powerful mathematical approach is 
based on the use of constraints. For example, in deconvolving a far-field image 
a good constraint would be to define with high precision the cell membrane of a 
cell that is stained with a dye and is being imaged by a lens. By precisely 
defining the position of a cell membrane or a portion of the cell membrane we 
can precisely define where the staining in the image is confined and beyond 
which point or points there is no staining and its associated optical 
phenomenon. Such a constraint would give many deconvolution algorithms a 
powerful advantage. Nonetheless, even though the idea is mathematically a 
powerful concept [Carrington et al, Science 268, 1483 (1995)], it is seriously 
limited in far-field optics by the inability to obtain a constraint that is better 
than the optical resolution. 

•III. State of Prior Art 

No one has previously attempted to incorporate such near-field optical data and 
other scanned probe microscope data such as that which is obtained from 
atomic force microscopy to the problem of providing constraints in the 
deconvolution of far-field optical and other far-field imaging techniques. 

IV. Summary of the Invention 

A method for deconvolving far-field optical images beyond the diffraction limit 
by using near-field optical and other scanned probe imaging data such as that 
which can obtained from atomic force microscopy on a region of the far-field 
data set in an integrated and inter-digitated way inorder to provide powerful 
and new constraints for the deconvolution of far-field data sets to resolutions 
beyond the diffraction limit limit of the lens that is being used or in the case of 
non-linear optical imaging or other microscopies to resolutions beyond that 
which is achievable with these microscopies. 



V. Description of the Invention 

The present invention incorporates data that has never been incorportated 
previously to try and resolve these issues of far-field imaging. This data comes 
from near-field optical microscopy and its scanned probe cousins such as 
atomic force imaging (AFM) and presents the far-field microscopist with 
constraints that will allow for improving dramatically the far-field imaging of 
all forms of far-field microscopy both linear and non-linear optical microscopy 
and even those microscopies that use particles rather than electromagnetic 
radiation. It also allows for using one form of scanned probe microscopy to 
deconvolve another and thus improve the scanned probe microscopic resoluton 
before this data is used in this invention as described below. 

For such an approach it is essential to obtain the near-field optical or other 
scanned probe imaging data in a way that is fully integrated with the far-field 
data set. In one emulation of this invention, in which far-field optical 
microscopic data is to be deconvolved, to achieve the required full integration 
of the data sets one useful approach is to use a charge coupled device (CCD) to 
record the far-field optical image. The near-field optical and AFM or other 
scanned probe data can be related directly to the exact digital pixel of the 
far-field data that is corresponding to the associated scanned probe pixel. 
Furthermore, it is additionally useful if the scanned probe data sets that are to 
be used in the deconvolution can be obtained in simultaneous channels. This 
can be done for example with a tip that is multifunctional such as a tip that is 
both a subwavelength light source and an AFM sensor [K. Lieberman, et al, 
Rev. Sci. Instr. 67, 3567 (1996)] that can be used in contact or near-contact 
with a surface of the specimen that is being imaged. 

However, even before the far and near-field imaging process is begun, the 
image of the subwavelength light source of known dimension can on the CCD 
is a measure of the point spread function (PSF), which is the lens function that 
convolutes with the functional representation of the sample to give the blurred 
image that is the far-field image with its associated diffraction and other 
problems mentioned in the Background section of this patent. Alternately the 
convolution effect of the lens can also be achieved if a known high resolution 
sample is imaged and the error between the real and the ideal image is 
represented as a blurring function introduced by the lens. Obviously, if the 
imaging task is fluorescence then the high resolution test object will have to be 
similarly fluorescent, thus the first task in this method is the PSF Of the lens. 

The next step is to record with the CCD the far-field image. Subsequently, 
super-resolution optical data can be recorded at specified points on the sample 
surface. Examples of such points can include: an exact point at which the 
optical contrast in a sample terminates, i.e. defining the edge of a sample to 
much better than the optical resolution if the far-field image is an optical 
image; or defining the x, y and z point or voxel at which there is a contrast 
change and relating this point to another point of contrast change in the sample. 
In terms of the relation of two points of contrast change in the sample, these 
could be at different planes as defined by the lens in the far-field. The scanned 
probe data, through the simultaneously recorded AFM, can provide not only the 
xy separation of the two points but also the Z separation at a resolution that is 
better than any optical approach such as confocal microscopy. 

The exemplary constraints listed above or for that matter any constraints from 
scanned probe technology have never been used in this cross-fertilization mode 



with far-field optical microscopy or for that matter any far-field microscopic 
technique. In addition such cross-fertilization has not been used between 
scanned probe techniques, i, e. to use near-field optical data as a constraint at 
deconvolving atomic force microscopy data or the reverse. With regard to this 
latter mode of deconvolution there is quite a bit of synergism between, for 
example, the near-field optical and the AFM since the functional dependence of 
the decay of the effect, as a function of probe sample separation, is 
near-exponential for the near-field optical and occurs over a much shorter 
distance for the simultaneously recorded AFM technique. In essence then the 
scanned probe microscopy synergism can first be used to improve the scanned 
probe microscopy data and then that data can be applied as a constraint to the 
far-field microscopy deconvolution in question. It is also important to note at 
this point, that, the order of the procedures listed in this section is not a critical 
part of the invention, any combination of the order of the steps or partial 
combination of steps constitutes this invention. For example, if the near-field 
optical data is not used to deconvolve to higher resolution the atomic force 
microscopy data, it, and the atomic force microscopy data could still be used to 
deconvolve the far-field data. 

The CCD mentioned above is a most useful method to obtain the digitized 
image of the far-field but this can also be accomplished with confocal 
microscopy. In the case of far-field optical microscopy it should be noted that 
there could be innovative ways to record the confocal data without any 
confocal aperture. For example, if there is a film of material that could produce 
a non-linear optical signal known as second harmonic generation (SHG) this 
could be used in an innovative way as part of this invention to record a 
confocal image. In this case the light from the plane of focus would be focused 
by the lens onto a film, such as a plastic film of purple membrane that produces 
SHG [Z. Chen et al, Applied Optics 30, 5188 (1991)], with an intensity that is 
higher than from any other plane in the sample that is being focused by the 
lens. Thus such a film could be used to replace the confocal pinhole by using a 
film that would produce a second harmonic signal only where there is a point of 
light from the plane of focus in the sample and thus such a film would act as a 
parallel filter for light from the plane of focus in the sample. This could be 
used together with an appropriate filter after the film to remove the 
fundamental wavelength that was illuminating the sample and only passes the 
SHG to the detector which could be a CCD rather than a single channel 
detector that is normally part of a confocal set-up. This could be done with 
SHG or other non-linear optically active films. 

VI. Advantages Over Prior Art 

Scanned probe microscopy data has not been used as a constraint in 
mathematical constraint algorithms to deconvolve far-field optical images. In 
addition, the use from multi functional scanned probe microscopy of one 
parameter, such as near-field optical data, to deconvolve another parameter 
such as atomic force microscopy data has also not been applied. The advantage 
over prior art arises from the increase in spatial resolution that this approach 
achieves. 

VII. Applications 

Methodologies for increased spatial resolution always open new doors in 
science and technology. Look at the revolution that was caused by the 
introduction of the electron microscope. 



To test the essence of this invention we have performed a calculation on a 
model far-field optical data set. In Figure 1 there are four parts. Starting from 
the top of the figure is a model object. When the object is imaged by a lens with 
a known point spread function the blurred image, which is the second from the 
top in Figure I , results. When this image is recorded by a CCD there is further 
blurring due to the pixel character of the CCD. The last image in this figure is a 
result of a standard deconvolution algorithm without the imposition of the type 
of constraints that is central to this invention in which a new approach to 
provide constraints is described. In Figure 2 the same object, lens, and CCD are 
assumed but in the deconvolution algorithm just 4 points are given with the 
resolution of near-field optics. These points are, in going from left to right in 
the model object, the first, second, third and fourth alterations in contrast. The 
results of using such constraints is seen in the vastly improved quality of the 
deconvolved image. 



Figures 



Figure 1 A model object (first image) imaged by a lens (second image), 
recorded on a CCD (third image) and deconvolved without constraints. 

Figure 2 The same as Figure 1 but the deconvolution was performed with 
constraints. 
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1 . A method for deconvolving far-field optical microscopic images to a level of 
resolution that has never been achieved previously by introducing scanned 
probe microscopy data in an interdigitated, integrated fashion with confocal or 
charge couple device imaging fashion such that every pixel or voxel from the 
super-resolution scanned probe microscopy data set is directly related to an 
image pixel or voxel in the far-field data set and then using the scanned probe 
data as a mathematical constraint to deconvolve the far-field image 

2. A method that uses multifunctional scanned probe microscopy data so that 
one scanned probe microscopy parameter can be deconvolved by integrating 
data from another such as, but.not excluding other combinations, deconvolving 
an atomic force microscopy image with a near-field optical microscopy image 

3. The method as in claim 1 in which the procedure in claim 2 is used in 
combination with the method in claim I. 

4. A method to obtain the point spread function, which is a crucial step to the 
deconvolution described in these claims, by using a known high resolution 
sample either fluorescent or non-fluorescent, which is imaged and the error 
between the real and the ideal image is represented as a blurring function 
introduced by the lens and is representative of an accurate point spread 
function. 

5. A method to obtain a digitized image of the optical far-field in addition to 
simple confocal or charge coupled device imaging that can be used for 
integrated near-field and far-field imaging that uses a film of material that can 
produce a non-linear optical signal known as second harmonic generation in 
which light from the plane of focus of the lens would be focused by the 
far-field lens onto a film, such as plastic purple membrane films, with an 
intensity that is higher than from any other plane in the sample that is being 
focused by the lens and thus such a film could be used to replace the confocal 
pinhole in confocal imaging by using a film that would light up only where 
there is a point of light from the plane of focus in the sample and thus such a 
film would act as a parallel filter for light from the plane of focus in the sample 
with this light being imaged through a filter for the fundamental illuminating 
frequency onto a charge coupled device. 

5. A device that collects in an integrated, correctable fashion that as a result 
allows for deconvolving far-field optical microscopic images to a level of 
resolution that has never been achieved previously by introducing scanned 
probe microscopy data in an interdigitated, integrated fashion with confocal or 
charge couple device imaging such that every pixel or voxel from the 
super-resolution scanned probe microscopy data set is directly related to an 
image pixel or voxel in the far-field data set and then using the scanned probe 
data as a mathematical constraint to deconvolve the far-field image 

6. A device that collects in an appropriate integrated fashion multifunctional 
scanned probe microscopy data so that one scanned probe microscopy 
parameter can be deconvolved by integrating data from another such as, but not 
excluding other combinations, deconvolving an atomic force microscopy image 
with a near-field optical microscopy image or vice versa. 



0 'A ( 



7. A device as in claim 5 that also has the capabilities of claim 6 

8. A device to obtain the point spread function, which is a crucial step to the 
deconvolution described in these claims, by using a known high resolution 
sample either fluorescent or non-fluorescent, which is imaged and the error 
between the real and the ideal image is represented as a blurring function 
introduced by the lens and is representative of an accurate point spread 
function. 

9. A device to obtain a digitized image of the optical far-field in addition to 
simple confocal or charge coupled device imaging that can be used for 
integrated near-field and far-field imaging that uses a film of material that can 
produce a non-linear optical signal known as second harmonic generation in 
which light from the plane of focus of the lens would be focused by the 
far-field lens onto a film, such as plastic purple membrane films, with an 
intensity that is higher than from any other plane in the sample that is being 
focused by the lens and thus such a film could be used to replace the confocal 
pinhole in confocal imaging by using a film that would light up only where 
there is a point of light from the plane of focus in the sample and thus such a 
film would act as a parallel filter for light from the plane of focus in the sample 
with this light being imaged through a filter for the fundamental illuminating 
frequency onto a charge coupled device. 



